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ABSTRACT 

This  report  is  a  review  of  a  series  of  activities  held  in  the  Joint  Decision  Support  and 
Simluaton  Centre  in  November  and  December  2007.  This  report  accompanies  a  classified 
report  containing  more  details  on  the  outcomes  of  the  activities.  The  purpose  of  this  report  is 
to  capture,  in  an  unclassified  way,  some  of  the  process  lessons  learned  during  these  activities. 
While  many  of  these  lessons  are  not  new  it  is  hoped  recording  them  in  this  way  will  help 
prevent  mistakes  being  repeated  in  the  future. 
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Lessons  From  Selected  Joint  Decision  Support  and 
Simulation  Centre  Activities  (Nov-Dee  2007) 


Executive  Summary 

This  report  is  an  unclassified  accompaniment  to  a  classified  report  on  the  same  topic. 
Together,  they  describe  an  event  made  up  of  a  series  of  activities  held  at  the  Joint 
Decision  Support  and  Simulation  Centre  (JDSSC)  in  Fairbairn,  Canberra  during 
November  (Event  1)  and  December  (Event  2)  2007.  The  activities  were  classified,  but  it 
is  beneficial  for  the  management  of  similar  activities  in  the  future  if  the  unclassified 
methodological  insights  are  presented  in  an  accessible  way. 

The  event  comprised  a  one  day  Defence-only  'rehearsal'  activity,  and  a  three  day 
activity  which  included  other  Government  agency  participants.  Both  of  these  were 
preceded  by  several  days  of  set  up,  and  a  day  of  practice  runs  for  each  activity.  One 
further  event  (Event  3)  was  planned,  but  was  subsequently  indefinitely  suspended 
following  the  change  of  government  in  November  2007.  Most  of  the  observations  refer 
to  Event  2,  the  December  activity  because  it  was  the  larger  and  longer  of  the  two 
conducted. 

This  report  attempts  to  identify  and  learn  from  errors  as  well  as  acknowledging 
techniques  that  worked  well.  As  such,  it  is  not  intended  as  an  overall  assessment  of  the 
event.  Nor  is  the  report  a  comprehensive  review  of  techniques,  rather  it  provides  some 
pragmatic  information  on  methodologies  that  worked  and  those  that  did  not  work  in 
this  context.  The  degree  to  which  the  event  was  a  success  can  be  determined  from  the 
goals  and  outcomes  as  set  out  in  the  joining  instruction,  this  is  assessed  in  the  classified 
report.  It  is  important  to  note  that  this  event  aimed  to  look  at  command  and  control 
structures  associated  with  the  system  rather  than  the  explicit  details  of  the  system 
itself. 

A  wide  variety  of  lessons  are  identified  in  four  areas:  planning,  game  play,  data 
capture  mechanisms  and  analytical  tools.  These  lessons  range  from  agreeing  goals  with 
clients  and  choosing  scenarios  that  appropriately  use  the  capabilities  to  room  set  up 
and  having  effective  facilitation. 
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1.  Introduction 


This  report  is  an  unclassified  one,  but  is  closely  associated  to  a  classified  report  on  the  same 
topic  [1],  Together,  they  describe  an  event  made  up  of  a  series  of  activities  held  at  the  Joint 
Decision  Support  and  Simulation  Centre  (JDSSC)  in  Fairbairn,  Canberra  during  November 
(Event  1)  and  December  (Event  2)  2007.  The  activities  were  classified,  but  it  is  beneficial  for  the 
management  of  similar  activities  in  the  future  if  the  unclassified  methodological  insights  are 
presented  in  an  accessible  way. 

The  event  comprised  a  one  day  Defence-only  'rehearsal'  activity,  and  a  three  day  activity 
which  included  other  Government  agency  participants.  Both  of  these  were  preceded  by 
several  days  of  set  up,  and  a  day  of  practice  runs  for  each  activity.  One  further  event  (Event  3) 
was  planned,  but  was  subsequently  indefinitely  suspended  following  the  change  of 
government  in  November  2007.  Most  of  the  observations  refer  to  Event  2,  the  December 
activity  because  it  was  the  larger  and  longer  of  the  two  conducted. 

This  report  attempts  to  identify  and  learn  from  errors  as  well  as  acknowledging  techniques 
that  worked  well.  As  such,  it  is  not  intended  as  an  overall  assessment  of  the  event.  Nor  is  the 
report  a  comprehensive  review  of  techniques,  rather  it  provides  some  pragmatic  information 
on  methodologies  that  worked  and  those  that  did  not  work  in  this  context.  The  degree  to 
which  the  event  was  a  success  can  be  determined  from  the  goals  and  outcomes  as  set  out  in 
the  joining  instruction,  this  is  assessed  in  the  classified  report  [1],  It  is  important  to  note  that 
this  event  aimed  to  look  at  command  and  control  (C2)  structures  associated  with  the  system 
rather  than  the  explicit  details  of  the  system  itself. 


2.  Background 

DSTO  assisted  CDG  staff  in  the  design  and  implementation  of  a  man-in-the-loop  command 
and  control  wargame.  In  this  wargame  the  tactical  level  was  represented  by  simulation 
models.  The  participants  were  divided  into  four  groups  during  the  wargames,  representing 
different,  geographically  dispersed  organisations  within  a  C2  structure. 

A  team  of  four  DSTO  analysts  was  established  to  carry  out  the  task.  The  team  was  provided 
limited  guidance  on  the  activities  and  found  it  difficult  to  seek  further  information  or 
influence  decision  makers  (the  client)  who  seemed  to  change  the  scope  of  the  events  after  the 
initial  guidance  had  been  given.  The  goals  of  the  events  (and  hence  the  analysis)  were  never 
clearly  articulated,  in  fact  elements  of  the  sponsor  group  appeared  to  have  differing  objectives 
based  on  discussions  from  the  activities1. 

Event  1  was  intended  to  be  preparation  for  Event  2,  which  was  in  turn  preparation  for  Event 
3,  which  was  later  indefinitely  postponed.  Elowever,  it  would  be  unfair  to  simply  write  off 


1  Aspects  of  this  can  be  attributed  to  the  transition  of  the  sponsor's  area  from  one  part  of  Defence  to 
another,  with  a  resulting  flux  in  staff  and  guidance. 
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these  activities  as  purely  rehearsals  because  both  activities  had  significant  numbers  of  subject 
matter  experts  present  and  resulted  in  the  collection  of  significant  relevant  data  for  the  client. 

Additionally,  Event  1  aimed  to  look  at  the  issues  in  an  Australian  Defence  Organisation/ Force 
(ADO/  ADF)  only  context  and  Event  2  then  looked  at  them  in  a  whole  of  government  context. 
Event  2  therefore  included  some  non-Defence  participants.  Event  3  was  supposed  to  look  at 
the  issues  in  a  coalition  context  with  participants  from  various  nations  invited  to  attend. 

Some  aspects  of  the  activity  were  dictated  by  the  sponsor  group,  including  the  use  of  a  simple 
simulated  environment  to  immerse  the  players  in  a  set  of  scenarios.  These  scenarios  were  used 
to  create  the  context  in  which  the  C2  wargame  was  studied,  they  did  not  need  to  be  (and  were 
not)  high  fidelity  simulations  of  the  tactical  battle. 

Each  scenario  was  examined  by  proceeding  through  four  phases  which  are  discussed  in  more 
detail  in  various  sections: 

1.  An  introduction  to  the  scenario. 

2.  The  human  in  the  loop  simulation. 

3.  A  review  phase  done  in  the  play  groups. 

4.  A  facilitated  discussion. 

Both  of  the  events  used  a  senior  ADF  member  as  event  facilitator  and  were  held  at  the  JDSSC 
facility  in  Canberra. 

2.1  Event  1 

The  analyst  team  was  given  just  over  three  weeks  to  plan  for  the  first  activity  held  on 
12  November,  and  during  this  time  met  several  times  to  discuss  the  guidance  given  and 
determine  specific  aims  and  client  requirements  from  the  activity.  Given  the  short  timeframe 
for  planning  (the  analysis  team  was  formed  in  mid-October),  the  team  selected  an  analytical 
approach  that  would  effectively  capture  stakeholder  views,  questions  and  enhance 
understanding,  with  minimal  set  up  time  and  rapid  turn  around  on  results.  This  approach 
focused  on  capturing  the  results  of  a  facilitated  discussion  aided  by  information  gathered 
from  active  observations2  and  questionnaires. 

It  was  agreed  that  the  client  requirement  for  findings  to  be  delivered  as  "insights  rather  than 
as  definitive  specifications",  indicated  that  quantitative  analysis  methods  would  not  be 
necessary.  Instead,  recording  of  discussions  and  analysis  of  key  questions  and  conclusions 
would  provide  sufficient  detail  and  rigour  to  satisfy  the  needs  of  the  client. 

The  primary  aims  of  Event  1  were  to: 

•  Rehearse  the  proposed  approach  with  ADF  participants  prior  to  Event  2  which  would 
include  other  Government  agency  participants; 

•  Collect  information  relating  to  ADF / ADO  thinking. 


2  Active  observations  allow  the  observer  to  directly  and  interactively  engage  with  the  subjects  to  elicit 
information. 
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Therefore,  during  the  activity,  the  analysis  team  focused  on  identifying  key  areas  of  interest  to 
drive  Event  2  discussions  and  potential  obstacles  to  sharing  of  views  and  maintaining 
participant  engagement. 

2.2  Event  2 

In  preparation  for  the  3-5  December  activity  the  team  sought  feedback  from  the  client  and 
assessed  the  strengths  and  weaknesses  of  Event  1.  This  assessment  yielded  improvements  for 
the  planning  of  Event  2.  These  improvements  are  detailed  under  appropriate  headings  below, 
but  largely  involved  changes  to  the  analytical  and  facilitation  techniques  and  substantial 
changes  to  the  scenarios  being  used. 

The  longer  nature  of  Event  2  meant  that  more  scenarios  were  covered  and  more  discussion 
was  entered  into.  It  also  allowed  for  some  of  the  scenarios  to  be  replayed  with  altered 
assumptions.  Additionally,  the  whole  of  government  approach  created  a  different  dimension 
to  the  event. 


3.  Planning  Lessons 

The  compressed  planning  timeframe  allowed  prior  to  the  events  was  not  ideal  for  research 
and  preparation  of  analysis  techniques.  It  has  been  suggested  that  this  also  reduced  the  ability 
to  secure  the  desired  number  and  variety  of  participants,  particularly  given  the  proximity  to 
the  Christmas  stand-down  period  and  end  of  year  deadlines,  but  this  timeframe  was  dictated 
by  external  factors.  The  Department  of  Prime  Minister  and  Cabinet  did  not  send  their 
representative  due  to  circumstances  beyond  the  control  of  the  organisers. 

In  retrospect,  having  a  series  of  events  leading  to  an  event  (Event  3)  which  was  then  cancelled 
was  not  the  best  approach.  However,  the  need  for  rehearsal  and  practice  events  is  well 
established  and  desired  amongst  experimentation  practitioners.  The  ability  to  successively 
practise  for  what  would  have  been  the  final  event  would  have  certainly  proved  valuable  to  its 
smooth  running,  although  it  does  mean  that  each  event  needs  to  have  well  established  goals 
that  link  together  and  build  on  each  other  in  order  to  avoid  confusion  and  duplication. 


4.  Game  Play 

The  activity  used  a  simulation  tool  to  run  four  teams  of  players  through  a  scenario.  While  the 
players  were  asked  to  respond  to  scenario  developments  they  were  not  able  to  influence  the 
events  occurring  in  the  scenario,  which  were  completely  pre-determined.  This  was  consistent 
with  the  analysis  team's  goal  of  looking  at  command,  control  and  communication  issues  and 
not  the  events  themselves. 
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Prior  to  game  play  an  introduction  to  the  scenario  was  carried  out.  This  session  introduced  the 
participants  to  some  basic  concepts  required  to  effectively  participant  in  the  scenario  and  gave 
them  an  opportunity  to  clarify  issues  such  as  terminology. 

The  teams  were  required  to  communicate  with  one  another  in  order  to  transmit  and  receive 
knowledge  as  they  saw  fit.  These  exchanges  were  governed  by  a  C2  structure  which  was  the 
main  subject  of  the  analysis. 

4.1  Teams 

The  four  teams  consisted  of  3-4  members.  This  smaller  number  appeared  to  be  sufficient  to 
allow  internal  discussions  and  also  appeared  not  to  be  excessive,  which  was  a  risk  given  the 
limited  role  of  participants  at  various  stages  of  the  simulation.  After  the  first  activity,  it 
became  clear  that  there  was  room  for  improvement  in  encouraging  all  participants,  regardless 
of  rank  or  experience,  to  share  their  views  as  part  of  the  group  discussion.  For  Event  2,  this 
was  considered  a  particularly  significant  issue  if  invited  representatives  from  other 
Government  agencies  were  to  feel  comfortable  in  making  a  contribution  where  they  may 
otherwise  have  felt  inexperienced.  The  analysis  team  concluded  that  this  could  largely  be 
achieved  by  emphasising  this  aspect  of  the  facilitator's  role. 


4.2  Room  Layout 

The  room  layout  at  the  JDSSC  was  relatively  simple  and  is  shown  in  Figure  1.  The  middle 
section  was  used  for  briefings  and  discussions.  The  edges  were  divided  into  sections  for  each 
of  the  four  teams. 
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Figure  1:  Simplified  Room  Layout 


The  barriers  between  teams  1  and  2  (and  similarly  between  team  3  and  4)  were  not  sufficient 
to  prevent  teams  from  being  aware  of  what  other  teams  were  doing.  This  was  little  more  than 
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a  distraction  in  the  context  of  this  event,  but  could  present  a  significant  problem  in  a  different 
wargaming  context  (i.e.  when  there  is  competition  between  the  cells  to  "win"  the  wargame  or 
when  there  was  inappropriate  information  that  could  have  been  shared  this  way). 

The  open  plan  layout  meant  that  players,  visitors,  analysts  and  white  cell  members  were  able 
to  move  easily  from  team  to  team.  This  was  useful  for  analysts  and  the  white  cell,  but  possibly 
the  movement  of  players  and  visitors  was  a  distraction.  The  space  available  was  adequate  for 
the  approximately  30  people  involved  in  this  event,  but  would  be  uncomfortable  for  any 
larger  number. 

4.3  Scenario  Usage 

Scenarios  were  designed  to  allow  participants  to  experience  different  perspectives  during  a 
crisis.  The  idea  being  to  then  explore  the  implications  of  that  experience  with  the  group 
during  the  post-run  discussions. 

The  scenarios  required  each  of  the  teams  to  play  a  different  role  in  a  command  and  control 
network  in  response  to  scripted  information  received  about  the  unfolding  situation.  Scripts 
were  read  by  members  of  a  non-playing  cell.  This  scripted  information  was  injected  at 
predetermined  times  irrespective  of  what  actions  the  players  had  taken  previously.  These 
were  synchronized  with  events  occurring  in  the  simulation. 

The  team  members  defined  their  own  goals  surrounding  their  wargame  role  in  the  scenario, 
based  on  the  information  they  received  during  the  pre-play  briefing. 

The  scenarios  provided  by  the  clients  for  use  in  the  activities  were  not  endorsed,  but  were 
approved  by  CDG  for  the  purpose  considered.  The  scenarios  were  elaborate  for  the  purpose  of 
the  event  and  stretched  some  of  the  underlying  assumptions  about  capabilities  associated 
with  the  events.  However,  they  were  designed  with  a  focus  on  stressing  the  systems  being 
examined,  rather  than  being  concerned  with  realism  or  likelihood.  Further,  during  Event  1  the 
analysis  team  concluded  that  some  of  the  more  realistic  aspects  of  the  scenarios  needed  to  be 
changed  to  make  them  more  engaging  for  participants.  This  extra  complexity  also  added  to 
the  resources  required  to  generate  the  supporting  simulation  and  other  elements. 

4.4  Communication 

Communication  between  teams  and  with  the  white  cell  was  conducted  using  a  number  of 
tools  that  are  described  in  Section  5. 

The  C2  network  was  prescribed  to  reflect  candidate  vertical  and  horizontal  lines  of 
communication.  This  resulted  in  communication  being  controlled  to  only  allow  certain  teams 
to  talk  to  one  another  based  on  the  C2  model  being  explored. 

4.5  Situational  Awareness 

Situational  awareness  was  provided  to  the  players  via  the  Dynamic  Agents  Representation  of 
Networks  of  Systems  (D ARNOS)  tool.  This  was  used  to  display  a  map  of  the  operational  area 
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containing  all  elements  and  in  some  cases  a  3D  rendering  of  events.  Each  participant  was 
provided  with  a  monitor  that  allowed  them  to  view  DARNOS  and  slightly  alter  the  way 
information  was  displayed  based  on  their  preferences  (i.e.  zoom  in  or  out,  or  view  their  team's 
communication  logs  etc).  The  information  presented  to  each  team  was  based  on  the 
information  available  to  them,  which  was  often  different  to  what  was  available  to  the  other 
teams  and  was  not  a  complete  picture. 

This  provided  the  players  with  some  information  than  was  not  necessary  to  their  decision 
making,  however  this  information  did  increase  the  players'  sense  of  immersion  in  the  game. 
Further,  the  event  did  prove  to  be  a  useful  opportunity  for  the  simulation  designers  to  practice 
using  situational  awareness  tools. 


5.  Data  Capture  and  Communication  Mechanisms 

Both  phone  lines  and  headsets  were  made  available  for  communication,  and  all  verbal 
communication  was  captured  via  room  speakers.  Other  data  capture  mechanisms,  including 
video  and  audio  capture,  are  also  discussed  in  this  section. 

In  a  few  instances,  having  multiple  methods  did  create  some  confusion  with  occasional 
duplication  and  conflicting  messages  being  given  over  different  systems.  However,  these  were 
mostly  unrelated  to  game  play  and  could  have  been  overcome  with  additional 
training/  protocols  had  they  been  problematic.  For  instance,  a  protocol  for  who  was 
authorised  to  give  certain  pieces  of  information  could  have  been  used. 

5.1  Phone  Logs 

The  phone  system  used  Voice  over  Internet  Protocol  (VOIP)  technology.  All  phone  calls  were 
logged  and  recorded  in  the  standard  WAV  format3  which  were  saved  to  a  location  on  a 
shared  drive  when  the  phone  call  ended.  Calls  were  logged  using  a  naming  convention  that 
identified  the  source  and  receiver  of  the  call  as  well  as  the  time  the  call  was  made.  It  was  these 
logs  that  were  subsequently  used  to  generate  the  social  network  analysis. 

The  phones  were  set  up  in  a  way  that  dealt  with  engaged  lines  by  ringing  secondary  phones 
available  in  each  cell. 

5.2  Headset  Logs 

An  internet  voice  chat  system  was  also  used  in  parallel.  It  was  constantly  available  on  several 
channels,  although,  for  the  small  number  of  participants  and  the  relative  simplicity  of  their 
roles,  it  was  not  often  used,  and  probably  not  necessary. 


3  WAV  files  have  the  extension  .wav  and  are  common  audio  files  able  to  be  played  in  Windows  and 
other  platforms. 
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The  primary  function  of  the  headset  system  was  as  a  means  for  the  part  of  the  white  cell 
conveying  scripted  information  to  communicate  with  the  players. 

All  the  conversations  were  recorded  but  were  not  reviewed  in  detail  as  the  content  of  the 
white  cell  inputs  was  already  known  and  conformed  to  the  script  in  nearly  all  cases. 

5.3  Video  and  Sound  Capture 

Video  and  sound  data  was  recorded  from  various  angles  during  the  event.  The  data  was 
streamed  live  into  the  analyst  room  and  was  used  to  monitor  the  situation  from  there.  It  has 
not  been  reviewed  since  the  event,  but  could  potentially  be  used  to  verify  data  from  other 
sources. 

5.4  Note  Taking  Through  Observation 

In  addition  to  the  team  of  four  DSTO  analysts,  there  was  a  number  of  observers  present 
throughout  the  event.  This  meant  that  there  was  more  than  sufficient  resources  to  make  note 
taking  and  passive  and  active  observation  an  effective  method  of  collecting  data.  This,  along 
with  the  whiteboard  notes  taken  during  group  discussion,  was  the  source  of  many  of  the 
conclusions  [1]. 

There  were  some  issues  with  notes  taking  too  long  to  be  safe-handed  from  the  JDSSC  to  the 
analysis  team  at  Edinburgh  and  some  notes  were  accidentally  destroyed.  Care  should  be 
taken  to  manage  data,  especially  classified  materials. 

5.5  Facilitated  Discussion  Groups  -  White  Board 

After  the  teams  of  players  had  been  given  a  chance  to  back-brief  on  their  experiences  during 
the  wargame  a  facilitated  session  was  conducted.  This  was  designed  to  further  encourage  the 
sharing  of  knowledge  and  discuss  assumptions  and  challenges.  Facilitation  was  performed  by 
two  people;  one  senior  ADF  member  and  a  senior  member  of  the  analysis  team. 

The  facilitation  was  partially  aimed  at  guiding  discussion  to  explore  some  key  issues  of 
interest,  determined  by  the  client.  Flowever  it  also  allowed  for  the  raising  of  new  issues.  This 
was  to  be  the  main  source  of  information  for  analysis. 

During  Event  1  it  was  noted  that  strong  personalities  expressed  their  views  on  each  of  the 
analysis  questions.  The  facilitator,  who  often  agreed  with  this  view,  did  not  then  allow  for 
explorations  of  wider  issues,  or  dissenting  opinions.  Especially  during  Event  1  the  facilitator 
interpreted  some  of  the  questions  in  the  questionnaires  (see  Section  6.1)  too  literally  which 
added  to  the  confusion  over  the  event's  goals.  It  was  therefore  concluded  after  Event  1,  that 
the  facilitated  discussion  would  benefit  from  greater  focus  on  the  activity  objectives. 

This  was  achieved  during  Event  2  by  ensuring  that  all  participants  provided  a  back  brief 
initially  and  the  cross-questioning  then  allowed  broader  discussion.  Additionally,  participants 
were  frequently  reminded  of  the  objectives,  and  the  facilitator's  and  lead  analyst's 


7 


DSTO-TN-0845 


responsibility  to  halt  discussion  and  redirect  it,  particularly  when  it  became  entrenched  in 
tactical  detail,  was  reinforced. 

From  this  experience,  the  analysis  team  concluded  that  the  military  facilitator  had  the 
experience  and  authority  (through  his  rank)  to  guide  and  direct  the  conversation.  While  the 
lead  analyst's  role  is  to  make  sure  that  sufficient  depth  and  breadth  are  covered  to  provide 
sufficient  relevant  data  for  productive  analysis  of  the  activity.  When  these  two  people  work 
effectively  as  a  team  they  possess  all  the  necessary  criteria  for  successful  facilitation. 

The  analysis  team  agreed  that  characteristics  of  good  facilitation  practice  for  this  type  of  event 
include: 

•  Record  consensus  opinions  and  describe  issues  of  disagreement  on  the  white-board; 

•  Striking  the  appropriate  balance  between  the  strategic  and  the  tactical; 

•  Encouraging  views  from  all  participants; 

•  Having  a  clear  understanding  of  the  purpose; 

•  Minimising  destructive  interjections  from  senior  stakeholders,  which  can  very  quickly 
erode  atmosphere  of  free  exchange  of  ideas  and  questions; 

•  Have  and  keep  in  mind  well  defined  structure,  goals,  and  deliverables  of  activity  in 
larger  project  context; 

•  Minimisation  of  repetition  of  well-made  points  by  using  intermittent  "Conclusions" 
summaries  which  give  assurance  that  a  point  has  been  captured  and  we  can  move  on; 

•  Usage  of  common  language  and  definition  of  key  terms; 

•  Alternating  which  individual  or  group  is  invited  to  start  each  discussion; 

•  Facilitator  asking  participants  to  articulate  the  assumptions  behind  their  assertions, 
even  when  there  is  consensus; 

•  Facilitator  having  the  freedom  to  not  slavishly  follow  prescribed  structure  and  judge 
when  a  given  issue  has  been  sufficiently  covered  in  conjunction  with  analytical 
guidance; 

•  For  broader  purposes,  participants  should  be  encouraged  to  go  beyond  focusing  on 
what  they  do  and  don't  know  about  the  topic.  Rather,  they  should  be  encouraged  to 
focus  on  immersing  themselves  in  the  simulation  enough  to  consider  what  they  would 
need  to  enable  them  to  carry  out  their  role  effectively,  what  the  measures  of 
effectiveness  would  be,  and  what  the  risks  and  consequences  of  ineffectiveness  would 
be; 

•  Not  be  advocates  for  a  point  of  view. 


6.  Analytical  Tools 

While  facilitation  and  note-taking  are  important  skills  for  analysis  there  were  two  areas  of 
more  traditional  analytical  work  which  are  reviewed  in  this  section.  These  are  the  use  of 
questionnaires  and  social  network  analysis. 
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6.1  Questionnaires 

The  information  captured  in  discussions  was  augmented  by  a  questionnaire  which  was  filled 
out  by  each  team.  After  each  simulation  run,  participants  were  asked  to  remain  seated  in  their 
respective  cells  and  were  asked  to  consider  their  responses  to  a  list  of  questions  that  would  aid 
them  to  identify  key  issues  for  their  back  brief.  These  group  discussion  questions  were  drawn 
from  the  master  questions  list  provided  by  the  client.  This  approach  aimed  to  lay  the 
foundations  for  enhanced  participation  and  greater  diversity  of  input,  by  distinguishing 
between  cell  and  group  discussion,  and  giving  each  cell  an  opportunity  to  communicate  their 
experience  to  the  rest  of  the  group  in  turn. 

The  questions  aimed  to  encourage  innovation  while  setting  boundaries  relevant  to  the  client's 
needs.  As  such,  the  questions  were  open,  but  also  asked  the  participants  to  think  about 
particular  issues.  Organisational  psychologists  from  DSTO  were  used  to  review  the  questions 
to  help  ensure  they  achieved  these  goals. 

During  Event  1,  participants  appeared  to  have  insufficient  time  to  fill  in  the  questionnaires, 
and  displayed  a  preference  for  sharing  their  experiences  verbally.  Therefore,  Event  2  benefited 
from  replacing  the  questionnaires  with  longer  group  discussion  time  which  was  more 
carefully  guided  by  a  predetermined  group  of  discussion  questions. 

6.2  Social  Network  Analysis 

The  phone  call  logs  provided  a  nearly  complete  record  of  the  information  transfer  (the  subject 
of  interest)  between  the  teams.  The  phone  records  were  reviewed  in  near  real  time  to  construct 
diagrams  similar  to  the  one  shown  in  Figure  2. 

The  linkages  between  the  teams  were  based  on  the  number  of  phone  calls  that  were  made 
between  teams,  and  also  reflected  the  direction  that  information  was  transferred.  Some  phone 
calls  were  eliminated  because  no  information  was  transferred  (such  as  joke  calls  and  wrong 
numbers)  and  in  some  cases  information  was  transferred  in  both  directions  in  a  single  call. 
Information  was  not  assessed  for  quality,  depth  or  accuracy,  however,  a  more  detailed  study 
could  have  included  these  factors.  Neither  was  information  tested  for  uniqueness  and  the 
same  information  could  be  represented  multiple  times. 

The  diagrams  presented  a  simple  way  to  represent  a  large  amount  of  phone  call  data.  Figure  2 
shows  that  Team  1  is  providing  information  to  Teams  2  and  4  but  not  3.  Team  2  is  then 
passing  on  information  to  Team  3.  In  this  structure  Teams  1  and  3  as  well  as  Teams  2  and  4 
were  unable  to  communicate. 

MS  PowerPoint  was  used  to  generate  the  plots  and  MS  Excel  was  used  to  tally  and  record  the 
data.  A  simple  diagramming  tool  might  have  aided  the  process,  especially  if  the  diagram  was 
able  to  be  automatically  generated  from  the  data. 

The  use  of  this  technique  was  not  planned  prior  to  the  experiment  and  is  a  positive  example  of 
how  ad-hoc  analysis  can  produce  robust  results.  Its  purpose  was  to  demonstrate  that  the 
command  and  control  structures  being  imposed  were  being  followed.  It  also  provided  an 
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insight  into  the  amount  of  information  which  was  being  shared  between  the  groups  and 
between  which  groups  it  was  being  shared.  This  in  turn  could  be  used  for  further  analysis  and 
comparisons  regarding  such  things  as  expected  information  flows. 


Figure  2:  An  example  of  a  social  network  diagram  showing  the  transfer  of  information  between  teams. 
The  numbers  are  the  number  of  information  exchanges  that  occurred  in  each  direction.  The  thickness  of 
the  lines  are  proportional  to  these. 


7.  Administrative/Minor  Matters 


7.1  Civilian  Attire 

The  use  of  civilian  attire  during  Event  2  encouraged  equitable  input  from  civilian  personnel 
and  ADF  members,  regardless  of  rank  or  organisation. 

7.2  JDSSC  Space 

The  benefit  of  the  JDSSC  being  away  from  the  hectic  Russell / ADFHQ  environment  was 
observed.  There  was  only  some  attrition  of  participants  and  those  that  did  participate  were 
fully  engaged  in  the  activity  while  they  were  there.  The  number  of  people  attending  the  event 
(in  whatever  capacity)  meant  that  the  middle  section  of  the  decision  space  was  not  large 
enough  to  cater  for  them  all.  Subsequently,  the  whiteboard  was  not  visible  to  all  participants 
during  the  facilitated  discussions. 
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7.3  Hospitality 

It  was  suggested  that  instant  coffee  might  not  be  acceptable  to  some  international  participants. 
The  urn  proved  excessively  noisy  when  in  the  JDSSC  Decision  Space. 

It  was  also  mentioned  that  it  would  be  appropriate  to  break-up  the  week  by  organisation  of 
social  activities  in  the  presence  of  international  participants. 

7.4  Security 

A  number  of  participants  and  analysts  expressed  a  perception  of  poor  security.  This  most 
commonly  related  to  the  frequent  absence  of  the  site  security  guard  from  the  entry  point  desk. 
This  was  compounded  by  errors  and  significant  delays  in  processing  visitor  passes  and 
responding  to  queries. 

7.5  Ethics  Permission  Sheets 

Participants  were  required  to  give  permission  for  data  to  be  captured  in  line  with  DSTO 
policy. 

There  was  at  least  one  issue  with  the  wording  in  the  permissions  sheets.  It  is  recommended 
that  the  committee  responsible  for  creating  the  pro-forma  for  the  approval  forms  review  some 
of  the  ambiguous  terms. 


8.  Summary 

This  report  summarises  some  lessons  identified  from  a  classified  set  of  activities  in  an 
unclassified  way.  It  is  not  meant  as  an  instruction  manual,  merely  as  a  mechanism  of 
recording  these  lessons  for  possible  future  consultation  in  similar  areas. 
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